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ABSTRACT: The quality control (QC) of assembled modules is an essential process when constructing modular 
buildings such as hotels and hospitals. Defects that go undetected during module assembly may result in lost 
productivity in the form of unnecessary transportation, rework or project delays. OC has traditionally been 
performed using specialized tools and carried out a posteriori in an inspection station dedicated solely to this task. 
Nowadays, large-scale 3D metrology technology provides a more efficient alternative since it enables accurate 
measurements to be taken in situ. Additionally, mixed reality (MR) supports the immersive projection of 
information and guidance instructions. This paper introduces a proof of concept of a framework that combines 
industrial photogrammetry with the HoloLens 2 MR headset to assist with assembly and QC during the off-site 
construction phase of modular construction. Many tests were conducted in a laboratory and a factory setting to 
evaluate the systems user-friendliness and possible challenges associated with its future implementation. The 
experiments conducted confirmed that combining 3D metrology with MR offers an interesting solution for 
integrating QC into the assembly process. However, further work is needed to enhance the measurement workflow 
and optimize the measurement system 5 accuracy. 
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1. INTRODUCTION 


In modular construction, volumetric building units, or modules, are factory-built with almost complete interiors 
including plumbing, electricity, insulation and even furniture. They are then transported to the construction site for 
final building assembly, which basically involves stacking the pre-assembled modules together. The modules’ 
structure can be made of wood, steel or a combination thereof depending on the customer’s requirements and the 
final height of the assembled building. 


During module assembly in the factory, the control of key characteristics (e.g., overall dimensions, squareness 
error, parallelism of the ceiling and floor) is essential to ensure the module complies with the specifications and 
estimate the adjustment shims needed when stacking modules on-site. Traditionally, this process has been carried 
out manually using tools like measuring tapes and levels in the assembly stage and controlled a posteriori at an 
inspection station dedicated solely to this task. If necessary, adjustments and corrections are then made, which 
leads to lost productivity in the form of unnecessary transportation, rework or project delays. 


The advent of contactless 3D metrology provides a valuable alternative since it makes it possible to take accurate 
measurements in situ, while building information modeling (BIM) provides a 3D digital representation of as- 
designed buildings. This combination of technologies thus automates inspection and digitalizes QC information. 
QC automation in the prefabricated construction industry has been addressed in recent research. Bae and Han 
(2021) proposed a vision-based approach to off-site quality inspection that reconstructs 3D point clouds using a 
projector camera system and computes how much scans deviate from the virtual model to generate quality 
assessment error maps. Kim et al. (2019) proposed to use a registration-free mirror-aided laser scanning approach 
to inspect the dimensions and geometric requirements of planar prefabricated elements. Xu, Kang, and Lu (2020) 
used laser scanning reconstruction technology to inspect surface defects in prefabricated concrete elements. The 
information they collected during QC was then stored in accordance with the Industry Foundation Classes (IFC) 
standard and integrated in a BIM platform. 


In industry, advances in 3D measurement systems have made it possible to incorporate inspection in the assembly 
process, which is referred to as measurement-assisted assembly (MAA). The term MAA is used to describe any 
process that involves measurements being used to guide assembly and QC (Muelaner, Kayani, Martin, & 
Maropoulos, 2011) 


MAA was first introduced as a paradigm shift for the assembly of high-quality large-scale complex structures like 
aircraft frames to eliminate the monolithic jigs and manual specialized tools that were usually involved in large 
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flexible-component assembly (Maropoulos, Muelaner, Summers, & Martin, 2014). This paradigm shift was 
motivated by advances in large-scale 3D metrology systems that made it possible to take measurements during 
fabrication or in situ. This is particularly beneficial because large-scale structures are often too big to fit into 
conventional measuring devices or be transported to calibration laboratories (Schmitt et al., 2016). While the 
construction industry requires less accuracy than the aerospace industry, the same concept can be used to integrate 
QC in the assembly process of modular and prefabricated structures. 


On the other hand, workers need to keep their hands free during assembly to move around and carry their tools. 
Augmented reality (AR) and mixed reality (MR) technology can make a significant contribution here. They merge 
computer-generated information with real-world sensations using a device (e.g., a head-mounted screen, a 
projector, or a tablet) that provides an immersive user experience and eliminates the need to constantly look at 
fixed screens for information. In the case of MR, the user can interact with the virtual objects (Peddie, 2017). MR 
can be seen as an evolution of AR that has been made possible by technological advances in sensors and imaging 
techniques (Park, Bokijonov, & Choi, 2021). 


Various studies have evaluated applying AR and MR to assembly tasks and inspection processes. Qin et al. (2021) 
investigated whether it was possible to use head-mounted AR displays for wood frame assembly tasks. Ahn, Han, 
and Al-Hussein (2019) proposed to use a projection-based AR system to provide workers with visual guidance 
during manual panel assembly. Their system projected as-designed models (panel drawings) into the assembly 
station. Kwiatek et al. (2019) demonstrated that using a mobile AR application in conjunction with 3D scanning 
during pipe section assembly and inspection improved productivity, reduced the amount of work that needed to be 
redone, and enhanced workers’ spatial skills. Talamas (2017) evaluated using MR interfaces to automate the 
metrology process flow for in-line assembly process inspection and found that each volunteer made fewer errors 
when using the MR interface than paper or laptop instruction guides. 


The aim of this paper is to propose a proof of concept of a framework that combines 3D measurement technology 
and MR to integrate QC in off-site module structure assembly. The purpose is to enhance productivity during the 
assembly process, provide more accurate measurements, and ensure quality output traceability. 


2. MATERIALS AND METHODS 
2.1 Measurement Equipment 


In this research, we focus on assembling a cuboid wood frame structure formed of a floor, four walls and a ceiling. 
We suppose that the quality of these six parts was previously controlled. To control the quality of the assembly 
process, we measure the 3D position of a set of points that will make it possible to control the key characteristics, 
(KCs) the overall dimensions, squareness error, parallelism of the ceiling and floor, etc. These positions will then 
be compared to the data represented in the computer-aided design (CAD) model. Fig. 1 illustrates the wood frame 
structure and the set of control points. 


Fig. 1: Wood frame structure 


1172 


SECTION E - ADVANCED TECHNIQUES FOR THE CONSERVATION AND MANAGEMENT OF BUILT ASSETS 


Photogrammetry is a technology that is based on the principle of optical triangulation. An element is positioned in 
3D space by at least two cameras that are used to identify targets from different viewpoints. The targets reflect 
infrared light. The cameras can then capture the position of the targets and position them in relation to the 
measurement system reference. Multiple targets can be perceived simultaneously, and their positions can be 
determined dynamically for real-time target tracking (= 10 Hz). Photogrammetry is used in many sectors for 
dimensional inspection. In our research, we use its tracking capability to provide 3D measurement data to assist 
with the assembly process. C-Track is a photogrammetry device from the company Creaform® that has a 
measurement volume of up to 16 m°. Its measurement range can be extended by combining up to four devices to 
form a measuring system around assembly stations. This eliminates the need to transport measuring equipment 
from one station to another. Additionally, C-Track can be integrated with portable scanners to probe or scan specific 
geometric elements as required. 


However, the retroreflective targets that are commonly used with C-Track, such as stickers and magnetic artifacts, 
are unsuitable for wood framing and cannot be accurately calibrated in the CAD model. To address this limitation, 
customized artifacts have been developed. The artifacts are designed to be easily attached to a wood frame. Each 
artifact is composed of three retroreflective targets (C1, C2, C3) to locate a point in 3D space. The targets have 
different spacing to be able to easily differentiate them. Fig. 2 shows a sample artifact developed for tracking the 
upper corner of a wall. 


Fig. 2: Sample artifact 


2.2 Measurement System Setup 


Prior to initiating the inspection process, certain preliminary operations must be conducted to prepare the 
measurement system. These operations are: environmental referencing, alignment (or registration), and tracking 
model creation. 


Environmental referencing: This step serves to establish a frame of reference for C-Track within the real 
environment. The working environment is identified using retroreflective targets (the targets shown in blue in Fig. 
3 (a)). These targets are then registered using the primary C-Track (if multiple C-Tracks are used) and exported as 
a reference file. Once this has been done, measurements can be taken by moving C-Track within the referenced 
environment. 


Alignment: To be able to compare measured point coordinates with the CAD model, the C-Track needs to be 
aligned within the 3D CAD volume. This involves creating reference entities that align the instrument in the 
metrology software workspace with the real-world instrument. We utilized the floor as a centerpiece. 
Three perpendicular planes were probed and used to define the measurement coordinate system (as illustrated in 
Fig. 3 (b)), which was then aligned with the CAD model. Creaform’s HandyPROBE portable probe was used with 
C-Track for this alignment process. 


Tracking model creation: The tracking model refers to the collection of retroreflective targets that the C-Track 
system dynamically tracks. To create this model, the targets comprising the tracked artifacts are registered using 
the C-Track. Fig. 3 (c) illustrates this process. The acquired targets are then exported as a text file for future use 
during the inspection process. 
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(b) Aligning the measurement 
coordinate system with the CAD model 


(c) Creating the tracking model 


Fig. 3: Measurement system setup 


2.3 Data Processing 


The raw data collected by C-Track has to be processed and interpreted in order to extract KCs and interpret the 
measurements in line with the technical specifications. VxElements® software was used with C-Track for data 
acquisition. For data post-processing, inspection software offers a wider range of tools. In this project, 
PolyWorks Inspector® was used to extract dimensions from a CAD model, import measurement data and compare 
the measurement data with the CAD data. Moreover, macro scripting was used to automate the dynamic 
measurement process workflow. 


For each artifact (see Fig. 2), C-Track provides the (x, y, Z) positions of the three targets (C1, C2, C3). It is thus 
possible to create a local coordinate system (£) attached to the artifact. Once the geometry of the artifact is known, 
the position of a point of interest (for example, an upper corner of a wall) can be determined in the local coordinate 
system. Geometric transformation makes it possible to determine the point of interest’s position in the global 
coordinate system (the measurement coordinate system). This logic is computed via macro scripts in 
PolyWorks Inspector®. The detailed method is explained below. 


The first step is to identify the targets, which involves associating them with each point of interest and 
distinguishing each artifact’s different targets. To identify the targets that correspond to each artifact, we calculate 
the Euclidean distances between the nominal position of the corresponding point of interest and the targets. The 
three targets that lie within a sphere of radius 10 cm around a point of interest are considered the targets associated 
with the relevant artifact. If one or more targets are missing, an error message is displayed. This approach is based 
on the assumption that the part (the wall, for example) is positioned roughly at its nominal (CAD) position. Once 
the three targets associated with each artifact have been identified, the three targets are distinguished by comparing 
the distances between the targets based on the artifact’s geometry. Equation 2.1 provides the vector calculations 
for creating a local coordinate system £ (tp, Jy, ke) around each artifact using the positions of the three targets (C1, 
C2, C3). 


i= Cz — Cy 
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The position of the point of interest in the global coordinate system g, P, = (xg, Yo Zg), is calculated by geometric 
transformation from the position of the point of interest in the local coordinate system £ (Pp = (xp, ye, ze)). This 
transformation is represented by Equation 2.2: 
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2.2 


Pg = Tye Pe 


where T, is the transformation matrix from the local coordinate system (£) to the global coordinate system (g). 
To represent this transformation using homogeneous coordinates, Equation 2.2 is equivalently expressed as 
Equation 2.3: 


Xg M11 M12 M3 ty] ~% 
Yal _|%21 T22 123 ty! JM 23 
Zg T31 T32 «+133 tz| |Z 

1 0 0 0 1 1 


Equation 2.4 donates the rotation matrix: 
Ta M2 113 x 
T21 T22 123| =|yīg vig Ykg 2.4 


The translation vector corresponds to the coordinate of the origin of the local coordinate system in the global 
coordinate system as donated by Equation 3.1: 

ty xclg 

le] =4yclg 2.5 


2.4 User Interface 


As mentioned above, this project uses MR to project dynamic measurements for a user during wood frame structure 

assembly. The HoloLens 2 headset is a completely standalone head-mounted display (HMD), which means that it 

doesn’t need to be connected to a separate computing device. In addition, it is MR-based, which means that it 

enables users to interact in real time with digital content that is superimposed on the real world. The content takes 

the form of holograms, and the holograms interact simultaneously with the user and the real world. Furthermore, 

HoloLens 2 enables users to interact with holograms using voice commands, hand gestures or eye movement. 

What’s more, the technology makes it possible for two users in different locations to see what the other sees, which 
enables one to guide the other through a process or simply interact with the world the other sees. This feature has 
the potential to make remote collaboration easier, more efficient and far more interactive. In addition, an MR plug- 
in exists for PolyWorks Inspector that makes it possible to manipulate an inspection project directly on HoloLens 2. 
Macro scripting can also be used to customize the user interface displayed by HoloLens 2. 


The user interface must provide the operator with the values of the KCs measured to guide the adjustments to be 
made and ensure the geometric quality of the assembly. In addition, it must enable the operator to perform certain 
control commands, such as exporting results and navigating between different inspection stages. 


The user interface proposed has two components (Fig. 4): (1) the menu or toolbar, which is composed of 
three buttons that are each associated with a command Button 1 launches dynamic measurement, Button 2 exports 
the measurement results, and Button 3 shows or hides the CAD hologram; and (ii) annotations to display the 
measured value’s deviation from the nominal value along the x, y and z axes. The annotations are displayed at 
the nominal position of the corresponding point of interest and change color depending on whether the measured 
value is within or outside of the tolerance interval. Note that the functionalities of the PolyWorks AR plug-in for 
HoloLens 2 were used to align the CAD hologram with the real environment. 
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Data processing and real-time annotation updating 


BB Test18 - Bloc-notes - o -Aa | 
Fichier Edition Format Affichage Aide 
Open the Align the CAD feta -mes-",0.043504,0.717059,489.953902 
a ‘ = “pt2 -mes-",733.128504,0,302184,489,561147 
inspection hologram with the Toolbar 
project real environment | 


Stop dynamic measurement and export the 
results 


Fig. 4: User interface created for the HoloLens 2 headset 


3. LABORATORY TESTING 


In order to conduct tests in a laboratory setting, a scaled-down module had to be constructed that adhered to spatial 
and ergonomic limitations. The scaled module replicates a typical wood frame structure and is around 1/8th the 
size of an actual module. The module was designed using CAD software. Dimensional and geometric data was 
present in the CAD model and served as reference information for the nominal (as-designed) model. The measured 
data was subsequently compared with this nominal data. 


3.1 Experimental Setup 


The floor was placed on a granite surface plate and secured to the table with a weight to prevent it from moving 
(see the picture on the right in Fig. 5). Next, we acquired the reference targets, performed alignment and acquired 
the tracking model as explained in § 2.2. 


In an industrial context, adjustable braces are used while assembling walls. A turnbuckle system was built to 
replicate this for the experiment and allow the operator to easily adjust the position of the wall while tracking the 
position of the point of interest (see the picture on the left in Fig. 5). 


= - >am 
Point 2 


Fig. 5: Experimental setup 
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3.2 Gage Repeatability and Reproducibility Study 


A measurement may be influenced by various sources of variation during the inspection process, which results in 
there being uncertainty associated with each measurement result. Measurement uncertainty is a quantitative 
assessment of the unreliability associated with a measurement result based on probability distributions. The 
dispersion of a set of measurements of a quantity can be characterized using the estimator of its standard deviation, 
which is also known as standard uncertainty (0). 


Studying the overall uncertainty of the measurement system and evaluating whether the system is able to accurately 
detect quality defects requires more in-depth study that is beyond the scope of this project. In this study, we intend 
only to evaluate the amount of variation in the measurement data that is attributable to the measurement system in 
the configuration proposed. Measurement system variation consists of two important factors, repeatability and 
reproducibility (R&R). Repeatability is related to equipment variation, whereas reproducibility is related to 
inspector or operator variation. Measurement system variation can be assessed by conducting a Gage R&R study, 
which involves data being collected by having multiple operators measure the same set of parts in a random order. 
Several methodologies can be used for statistical analysis of the data obtained from the Gage R&R study. We chose 
to use the ANOVA method, which breaks down the sources of measurement system variation as follows:(1) part- 
to-part: variation originating from the parts being studied; (2) reproducibility: variation originating from the 
operator(s); (3) operator/part: variation arising from the operator(s) interacting with the parts; and (4) repeatability: 
variation that originates from the measuring system and cannot be attributable to other sources of variation. The 
purpose of measuring multiple parts is to evaluate manufacturing method variation, which is also beyond the scope 
of our research. Therefore, only one part is measured in this study and, thus, variation sources (1) and (3) listed 
above (which are usually provided by ANOVA) are not taken into consideration in our analysis results. 


3.3 Test Sequence 


Two operators were asked to repeat the positioning of a wall while tracking the position of the wall’s two upper 
corners. An operator begins by wearing the MR headset and opening the project. The interface displays a hologram 
of the CAD file with the wall in its nominal position along with a three-button menu, the latter of which is described 
in § 2.4. The operator then positions the wall approximately in its nominal position and attaches it to the floor and 
the adjustable assembly brace system. They then fasten the artifacts to the top corners of the wall. Afterwards, they 
start dynamic measurement by pressing Button 1. The annotations are displayed to indicate each point of interest’s 
positioning error on the x, y and Z axes. 


The operator begins by adjusting the wall’s position along the x-axis in accordance with the deviation values 
displayed in the annotations for Points | and 2 (see Fig. 5). Once the x coordinate is within tolerance, the operator 
adjusts the position of Point | along the y and z axes by adjusting the adjustable brace. When Point 1’s position 
is within tolerance along all three axes, the annotation changes from red to green. The operator then moves on to 
Point 2 and adjusts its position along the y and z axes. Once both annotations are green (see Fig. 5), the operator 
stops dynamic measurement by pressing Button 2, which automatically exports the values to a text file. Each 
operator repeated the sequence 37 times, with six measurements captured each time: Pointl _x, Pointl_y, Pointl _z, 
Point2_x, Point 2_y and Point2 _z. 


3.4 Results and Discussion 


The data collected was analyzed using the ANOVA method. Table 1 indicates the repeatability and reproducibility 
standard deviation of x, y and z for Points 1 and 2. The precision-to-tolerance ratio P/T is expressed by 
Equation 3.1: 


60 


= —— 3.1 
USL — LSL 


P/T 


where USL is the upper specification limit, LSL is the lower specification limit, and ø is the standard deviation 
of the measurement error. 
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Table 1: Gage R&R results 
Point 1 Point 2 


x y Z x y Z 

Orepeatability 0.335 0.487 0.152 0.325 0.413 0.089 

Oreproducibility 9.000 0.019 0.028 0.000 0.237 0.029 
ORaR 0.335 0.487 0.154 0.325 0.476 0.093 
P/T 33.5% 49% 15% 33% 48% 9% 


The results show that the maximum total standard deviation is 0.487 mm. This means that 95% of the time, the 
measurement value varies by no more than 407 ota) = 1.948 mm (+26rota = +0.974 mm), which is below the 
maximum allowed tolerance range of 6 mm (+3 mm) indicated in the technical specification. However, for a 
measurement system to be considered “good”, its precision-to-tolerance ratio must be <10%. (Note that 
10%<P/T<30% is borderline, and P/T>30% is unacceptable). 


It should be noted, however, that system variation is influenced by the fact that the operator is asked to position 
each point within an interval of +1 mm. This can be confirmed by the fact there is less variation along the axis-z 
since this value is the least impacted by the positioning interval. In fact, when the operator adjusts the braces, the 
wall’s position along the z-axis almost doesn’t change. In addition, given the experimental setup, the y-axis 
corresponds to C-Track’s depth axis. A study of the effect C-Track’s depth of field has on measurement system 
variation showed that the depth of field has a significant effect on system repeatability (Emond-Girard, 2022). In 
all cases, the system repeatability error is greater than the reproducibility error, which means that the greatest 
source of variation is the measurement system itself, not operator manipulation. 


4. FACTORY TESTING 


In order to assess the user-friendliness of the proposed system and to highlight potential constraints linked to its 
use in an industrial environment, a test was carried out under real working conditions. Below is a description of 
the experimental setup and test procedure used, as well as the findings and observations noted following the test. 


4.1 Experimental Setup and Test Sequence 


A near-real-size module was designed to perform factory testing. The artifacts were also adjusted to a true scale. 
All system setup steps, including referencing the environment, aligning the measurement system with the CAD 
model’s global coordinate system, creating the tracking model, and aligning the CAD hologram with the real 
environment were first completed by the research team. Then, we explained to the operators assigned to take part 
in the tests how the system worked. Three (3) operators then worked together to pre-assemble the module and add 
the adjustable wall braces. One operator attached the artifacts and put on the MR headset to begin positioning the 
wall beginning with Point 1. Since the annotation displayed on the MR headset was small, the operator had to 
climb a ladder to read the value. The other two operators adjusted the position of the wall following the instructions 
given by the operator wearing the MR headset. The process was repeated until the position of Point 1 was within 
the desired tolerance range. The process was then repeated for Point 2 following the same procedure. Fig. 6 shows 
the test sequence. 
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4.2 Observations and Discussion 


The preliminary actions involved in referencing the environment can be quite demanding in an industrial context. 
The referencing targets that C-Track requires to locate itself may move due to vibration and operator movement. 
In addition, during the assembly stage, the floor often moved (drilling of the floor to fix the adjustable wall braces, 
operator movement, etc.), which made the proposed registration method (alignment of measured data and CAD 
data) not well suited to the real context. The operator who used the HoloLens 2 headset reported that it was easy 
to use and comfortable to wear and work with. However, during the test, some drawbacks were noted with the user 
interface, such as the size of the annotations, which was deemed to be too small. Also, we had to explain to the 
operator the orientation of the axes and how to interpret the values displayed in the annotation because the 
coordinate system axes and displacement vectors were not indicated. 


5. CONCLUSION 


In this research, we were able to achieve a proof of concept of a system that combines industrial photogrammetry 
and MR to assist operators during the assembly of a wood frame structure. Points of interests were tracked and 
compared to the nominal data presented in the CAD model. The deviation between the nominal value and the 
measured value was projected on the real assembly through the MR headset. The proposed solution also supported 
the documenting of measurement results. 


Although the Gage R&R study results showed that the measurement system varied less than the allowed tolerance 
range indicated in the technical specification, its precision-to-tolerance ratio is still higher than what is 
recommended. The error associated with artifact fabrication can contribute significantly to the system’s variation; 
thus, further research should work on optimizing the artifact and using the artifact’s as-manufactured geometry 
when computing the measurement data to minimize the amount of variation generated by the artifacts. Further 
research must be done to evaluate the measurement system’s overall uncertainty and validate the system’s ability 
to detect nonconformities. 


The test completed in the industrial context helped to identify some drawbacks related to the proposed solution, 
and, thus, this research serves as a guideline for potential future implementation of a quality control system that 
combines 3D metrology and MR for integration in an assembly process. 


The proposed measurement setup was complex and time-consuming, and keeping the reference targets stable 
seemed to be a struggle during factory testing. In order to use photogrammetry in in situ measurement-assisted 
assembly, fixed reference entity has to be integrated in the plant layout. Other large-scale technologies like indoor 
global positioning system (iGPS) could be investigated as an alternative type of measurement equipment. Also, 
improvements could be made to the user interface to provide the operator with more visual guidance and a more 
ergonomic way to display the measured values could be sought. 
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